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LESSONS  LEARNED  FROM  ET  DESIGN  PROCESS  FOR  ASAS/ENSCE 


INTRODUCTION 


The  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social 
Sciences  (ARI )  and  the  Army's  Project  Manager  Training  Devices  (PM 
TRADE)  are  responsible  for  a  research  and  development  program  exploring 
the  capabilities  of  Embedded  Training  (ET).  Embedded  training  is 
defined  as  training  that  is  provided  by  capabilities  designed  to  be 
built  into  or  added  into  operational  systems  to  enhance  and  maintain 
the  skill  proficiency  necessary  to  operate  and  maintain  the  equipment 
end  item.  The  objectives  of  the  program  are  the  development  and 
evaluation  of  procedures  and  guidelines  for  targeting  ET  needs  and 
implementing  ET. 

This  report  is  one  in  a  series  providing  input  into  the  develop¬ 
ment  of  ET  procedures  through  application  of  guidelines  for  exemplar 
systems.  This  report  covers  the  lessons  learned  obtained  from  appli¬ 
cation  to  the  U.S.  Army's  All  Source  Analysis  System  (ASAS),  and  its 
Air  Force  equivalent,  the  Enemy  Situation  Correlation  Element  (ENSCE). 
The  development  of  both  systems  is  being  performed  under  the  auspices 
of  the  Joint  Tactical  Fusion  Program  Office  (JTFPO).  ASAS  and  ENSCE 
are  Automated  Information  Systems  (AISs)  that  will  be  used  by  Army  and 
Air  Force  Military  Intelligence  (MI)  personnel  to  integrate  and 
disseminate  collected  information.  ASAS/ENSCE  will  be  fielded  in 
several  releases.  The  earlier  releases  will  contain  the  bulk  of  the 
operator  functions.  Later  releases  will  build  upon  this  foundation, 
and  will  occur  as  Pre-Planned  Product  Improvements  (P^I). 

ASAS/ENSCE  was  selected  for  examination  for  several  reasons. 

First  of  all,  as  an  AIS,  it  is  a  type  of  system  for  which  ET  is  par¬ 
ticularly  potentially  feasible  and  desirable.  It  is  a  computer-based 
system  on  which  the  training  for  the  system  may  be  delivered.  Second, 
ASAS/ENSCE  exhibits  certain  system  characteristics  that  are  not 
addressed  in  detail  by  the  existing  guidelines  for  the  development  of 
ET.  These  guidelines  are:  A  Procedure  for  Developing  Embedded 
Training  Requirements  (Roth,  et  al.,  1986)  and  Interim  Procedures  for 
Embedded  Training  (ET)  Component  Design  (Fitzpatrick,  et  al.,  1987). 

The  current  versions  of  the  guidelines  are  limited  because  they  are 
based  upon  work  done  on  systems  further  along  in  the  development 
process  and  dominated  by  proceduralized,  psychomotor  tasks. 

ASAS/ENSCE,  on  the  other  hand,  is  a  system  that  is  in  an  earlier  stage 
of  development.  Also,  the  tasks  associated  with  ASAS/ENSCE  have  an 
important  cognitive  component  in  that  the  user  is  constantly  making 
decisions  concerning  information  access  and  use.  Thus,  ARI  and  PM 
TRADE  felt  that  examination  of  a  system  such  as  ASAS/ENSCE,  in  light  of 
the  existing  ET  guidelines,  could  provide  a  basis  for  the  expansion  of 
those  guidelines  to  earlier  stages  of  development  and  to  more 
cognitively  based  systems. 

This  document  is  the  fourth  in  a  series  of  four  reports  resulting 
from  the  application  of  ET  guidelines  for  the  determination  of  ET 
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requirements  for  ASAS/ENSCE,  and  the  subsequent  development  of  a  design 
for  ASAS/ENSCE  embedded  training.  It  should  be  more  properly  stated 
that  most  of  the  work  for  this  effort  has  been  focused  on  the  ASAS 
component  of  the  system,  because  of  limitations  on  information  availa- 
ability  for  ENSCE.  The  first  report  generated  for  this  project, 
entitled  Preliminary  Task  Descriptions  for  ASAS/ENSCE  (Evans,  et  al., 
1987a),  contained  a  collection  of  operator  tasks  for  ASAS.  These  tasks 
included  these  for  the  six  functional  areas  that  will  be  represented  in 
the  target  release  for  the  system,  as  well  as  common  user  tasks  that 
will  also  be  supported  by  this  release  and  two  functional  areas  that 
will  be  supported  by  later  releases.  The  second  report,  entitled, 


(Evans,  et  al.,  1987b),  was  the  result  of  the  application  of  ET 


requirement  selection  procedures  as  presented  by  Roth,  et  al.,  1986, 
with  some  modification  to  the  ASAS  tasks  generated  for  the  first 
document.  The  third  report,  Preliminary  Embedded  Training  Design  and 
Integration  Concepts  for  ASAS/ENSCE  (Evans,  et  al. ,  1987c),  contained 
ET  design  guidance  input  for  the  ASAS/ENSCE  training  developers. 

An  original  i ntent  of  this  project  was  to  use  to  the  fullest 
extent  possible  the  guidelines  for  ET  design  that  are  being  developed 
by  ARI  and,  through  the  use  of  the  procedures  outlined  in  the 
guidelines  document,  determine  weaknesses  in  those  procedures.  The 
current  document  contains  the  lessons  learned  from  the  project  as  a 
whole  and  recommendations  for  modifications  of  the  existing 
guidelines . 

This  document  presents  the  methodology  used  for  this  project  and 
the  results  of  the  project.  There  are  two  types  of  outcomes  of  this 
project:  one  is  the  actual  products  of  the  project,  represented  by  the 

ASAS/ENSCE  task  analysis,  the  ASAS/ENSCE  ETRs,  and  the  ET  design  recom¬ 
mendations;  the  second  type  of  outcome  of  the  project  is  the  set  of 
lessons  learned  from  the  analytic  process  that  was  followed.  In  this 
report,  the  focus  is  on  the  second  type  of  outcome  and  the  implications 
for  the  ET  guideline  development  process. 

There  are  four  sections  to  this  document  in  addition  to  this 
Introduction.  Each  of  the  first  three  sections  deal  with  one  phase  of 
the  project.  These  three  phases  are  as  follows: 


1.  Task  analysis. 


» 


2.  The  selection  of  ETRs. 

3.  The  design  of  an  ET  component  for  the  ASAS/ENSCE  training 
system. 

All  three  of  these  sections  are  similar  in  format.  Each  section 
contains  a  statement  describing  the  task,  a  subsection  on  the  method¬ 
ology  employed  to  perform  the  task,  discussion  of  the  lessons  learned 
from  performing  the  task,  and  any  recommendations  generated  from  the 
lessons  learned.  The  final  section  of  this  document  is  a  summary  of 
the  lessons  learned  that  prevailed  through  all  phases  of  the  project. 
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ASAS/ENSCE  TASK  ANALYSIS 


Statement  of  the  Task 


The  Initial  task  that  was  undertaken  for  this  project  was  to 
develop  a  set  of  task  descriptions  for  ASAS/ENSCE  operator  tasks. 

These  tasks  were  limited  to  those  performed  on  the  Portable  ASAS/ENSCE 
Workstation  (PAWS)  and  covered  the  following  areas: 

1.  System  Supervisor  (SS). 

2.  Collection  Management  (CM). 

3.  All  Source  Analysis  (AS). 

4.  Target  Analysis  (TA). 

5.  Situation  Analysis  (SA). 

6.  Communications  Intelligence  (COMINT). 

7.  Electronic  Intelligence  (ELINT). 

A  list  of  Common  User  (CU)  Tasks  was  also  developed. 

In  addition  to  the  steps  in  performing  each  task,  information  that 
was  to  be  used  in  later  phases  of  the  project  was  also  included  for 
each  task.  This  information  included,  but  was  not  limited  to:  refer¬ 
ence  source  of  the  task,  equipment  used  to  perform  the  task,  and 
prerequisites  for  task  performance. 


Methodology 


Two  sources  of  information  were  used  to  generate  the  task  descrip¬ 
tions:  documentation  and  discussions  with  military  intelligence 

Subject  Matter  Experts  (SMEs).  The  documentation  that  was  examined  was 
as  follows: 

Classification  Title 

Confidential  Software  Operators  Manual,  Application  Software  Module, 

Situation  Analyst  Subsystem,  R1  Delivery  (u) .  Jet  Propulsion 
Laboratory,  2  April  1986. 

Secret  Preliminary  Cost  and  Training  Effectiveness  Analysis  (U) , 

Volume  II  (Draft),  XMCO,  30  November  1984. 
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Classification 


Title 


Confidential 

Confidential 

Confident ial 

Confident ial 

Confidential 

Confidential 

Confident ial 

Confident ial 

Confidential 

Confident ial 

Confidential 

Confidential 

Secret 

Secret 

Secret 

Secret 


ASAS/ENSCE,  R1  User's  CDR,  Day  1  Narrative  Outline  (U), 
Jet  Propulsion  Laboratory,  1-5  April  1986. 

ASAS/ENSCE,  R1  User's  CDR,  Day  1  Alphanumeric  Displays 
(U) ,  Jet  Propulsion  Laboratory,  1-5  April  1986. 


ASAS/ENSCE,  R1  User's  CDR,  Day  2  Narrative  Outline  (U), 
Jet  Propulsion  Laboratory,  1-5  April  1986. 

ASAS/ENSCE,  R1  User's  CDR,  Day  2  Alphanumeric  Screens 
(U) ,  Jet  Propulsion  Laboratory,  1-5  April  1986. 

ASAS/ENSCE,  R1  User's  CDR,  Day  2  Graphic  Displays  (U), 
Jet  Propulsion  Laboratory,  1-5  April  1986. 

ASAS/ENSCE,  R1  User’s  CDR,  Day  3  Narrative  Outline  (U), 
Jet  Propulsion  Laboratory,  1-5  April  1986. 

ASAS/ENSCE,  R1  User's  CDR,  Day  3  Alphanumeric  Screens 
(U) ,  Jet  Propulsion  Laboratory,  1-5  April  1986. 


ASAS/ENSCE,  Rl  User's  CDR,  Day  4  Narrative  Outline  (U), 
Jet  Propulsion  Laboratory,  1-5  April  1986. 

ASAS/ENSCE,  Rl  User's  CDR,  Day  4  Alphanumeric  Screens 
(U) ,  Jet  Propulsion  Laboratory,  1-5  June  1986. 

Situation  Analyst,  Functional  Area  Experts  Forum  (U), 
Joint  Tactical  Fusion  Program  Office,  14-16  August 
1984. 

All  Source  Functional  Area  Experts  Forum  (U),  Joint 
Tactical  Fusion  Program  Office,  14-16  August  1984. 

Situation  Analyst,  Functional  Area  Experts  Forum  (U), 
Joint  Tactical  Fusion  Program  Office,  10  January  1985. 

ASAS/ENSCE  Functional  Area  Expert  Forum  (U), 

Applications  Software  Module,  Communications 
Intelligence  Subsystem.  Jet  Propulsion  Laboratory,  26 
December  1985. 

ELINT,  Functional  Area  Experts  Forum  (U),  Joint  Tactical 
Fusion  Program  Office,  15  May  1985. 

ELINT  II,  Functional  Area  Experts  Forum  (U),  Joint 
Tactical  Fusion  Program  Office,  27  January  1986. 

ASAS/ENSCE,  1989  System  Requirements  Review  (with  Volume 
II)  (U),  Jet  Propulsion  Laboratory,  15-19  September 
1986. 
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Classification 


Title 


Secret 


ASAS/ENSCE,  1989  System  Requirements  Review,  System 
Security  (U),  Jet  Propulsion  Laboratory,  15-19  September 
1986. 


Secret 


ASAS/ENSCE  Functional  Capabilities  Document  (U),  Jet 
Propulsion  Laboratory,  7  December  1983. 


Confidential 


ASAS  Operational  and  Organizational  Plan  (0&0)  (U),  U.S. 
Army  Intelligence  Center  and  School,  18  July  1986. 


The  documents  were  read  and  initial  task  description  lists  were 
produced.  These  lists  were  then  submitted  for  review  by  SMEs  in  the 
military  intelligence  field.  It  was  found  that  the  SMEs  were  able  to 
offer  input  and  validation  only  for  tasks  that  were  at  a  non-detailed 
functional  level  since  the  operational  system  capabilities  and  user 
interface  were  still  being  modified  by  the  software  engineers.  In  some 
cases,  detailed  tasks  could  be  hypothesized,  but  the  SMEs  were  not  able 
to  attest  to  the  complete  accuracy  of  the  tasks. 


Lessons  Learned 


Six  problems  with  the  collection  of  task  data  during  early  system 
development  were  identified.  Two  of  these  problems  are  discussed 
concur rei.ll/ .  The  other  four  are  discussed  as  separate  topics. 


Uncertain  Completeness  and  Accuracy 


It  was  found  that  during  early  stages  of  system  development 
available  sources  of  task  data  contained  information  that  wa«=  either 
inaccurate  or  inconsistent  between  sources.  These  two  problems  are 
both  reflections  of  a  single,  underlying  cause,  namely  the  evolutionary 
nature  of  an  emerging  system.  What  is  true  for  a  system  at  one  point 
in  time  may  not  be  true  for  the  system  at  a  later  point  in  development. 
Documentation  produced  at  different  points  in  time  will  usually  reflect 
only  the  current  concept  of  the  system.  To  overcome  this  problem, 
training  developers  must  work  from  current  documentation  whenever 
possible,  relying  on  other  sources  to  fill  in  gaps  in  the  data  with  a 
notation  indicating  the  historical  placement  of  this  source  in  relation 
to  more  current  information.  It  must  also  be  realized  that  any  task 
list  generated  for  an  emerging  system  must  be  seen  as  correct  only 
within  a  specified  time  frame. 


Along  with  inaccurate  and  inconsistent  information,  it  was  found 
that  there  were  omissions  in  the  data  sources  concerning  tasks  that 
were  known  to  occur  in  the  ASAS/ENSCE  releases  under  examination.  It 
was  often  the  case  that  one  could  infer  the  existence  of  "  cet  of  tasks 
based  on  the  presence  of  a  menu  choice  shown  in  documentation  of  user 
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screens.  However,  the  documentation  did  not  indicate  the  user's  tasks 
after  the  selection  of  a  menu  choice.  Thus,  the  details  of  these  tasks 
could  only  be  assumed  and  left  for  modification  during  a  later 
iteration  of  task  list  development. 

Given  the  available  documentation,  it  was  found  that  in  many  cases 
tasks  could  not  be  generated  to  the  level  specified  by  the  document 
entitled,  A  Procedure  for  Developing  Embedded  Training  Requirements 
(Roth,  et  al.,  1986).  Initially,  the  intention  had  been  to  apply  the 
Embedded  Training  Requirements  (ETRs)  selection  procedures  to  the  task 
descriptions  generated  for  this  project.  However,  due  to  the  lack  of 
detailed  data,  it  became  apparent  that  these  procedures  were  not 
applicable  to  data  generated  at  an  early  stage  in  system  development. 

It  is  possible  that  the  existing  procedures  may  need  some  modification 
in  order  to  successfully  produce  ET  requirements  from  data  that  fit  one 
of  four  categories:  functional  data,  detail-deficient  task  data, 
detailed  task  data,  or  data  composed  of  a  mixture  of  any  or  all  of  the 
other  types. 


Lack  of  SMEs 

It  also  became  apparent  during  this  task  that  one  other  assumption 
mentioned  in  the  ETR  procedure  document  could  not  be  met  by  the  task 
data  derived  for  ASAS/ENSCE.  This  is  the  assumption  that  there  are 
SMEs  available  to  verify  the  tasks  and  each  task's  associated  ratings 
for  complexity,  criticality,  and  perishability.  In  order  to  have  tasks 
and  ratings  verified,  one  needs  to  have  access  to  an  appropriate  set  of 
SMEs.  An  SME,  in  the  context  of  training  development,  is  someone  who 
is  an  expert  performer  of  the  tasks  that  are  to  be  verified.  In  the 
case  of  a  computer-based  system  used  as  an  information  management  tool, 
an  SME  should  be  an  individual  who  is  both  knowledgeable  about  the 
tasks  involved  with  operating  the  computer  system  and  about  the  tasks 
involved  in  operating  within  the  environment  in  which  the  computer 
system  is  to  be  used.  As  a  nc.-'ly  emerging  system,  ASAS/ENSCE  is 
lacking  in  personnel  who  are  experts  at  operating  the  computer  system 
within  the  military  intelligence  environment  into  which  it  will  be 
placed. 


The  ETR  procedures  do  not  make  allowances  for  the  case  in  which 
there  is  no  SME  support  for  task  verification.  It  is  imperative  that 
the  importance  of  SME  input  into  the  task  derivation  process  during 
early  stages  of  system  development  be  examined  as  a  research  question. 
The  findings  of  such  research  can  then  be  used  to  expand  the  ETR 
procedures  to  account  for  cases  in  which  SME  support  is  not  available. 


System  Design  Coordination 

The  final  lesson  learned  during  the  derivation  of  the  ASAS/ENSCE 
task  data  concerns  the  need  for  schedule  correlation,  coordination,  and 
interaction  between  developers  of  the  operating  system  and  developers 
of  ET  for  that  system.  Training  and  system  developers  need  to  begin 
their  interactions  as  early  as  possible.  If  there  is  no  coordination 


and  interaction  between  these  two  groups  then  two  problems  occur.  The 
first  is  that  without  early  interaction,  training  developers  have 
little  or  no  input  as  to  the  impact  of  the  expected  training 
requirements  upon  the  development  of  the  system.  Training  estimates 
must  be  made  early  in  order  to  impact  the  system  design.  The  current 
means  for  making  those  estimates  for  ET  are  unsatisfactory  in  that  the 
existing  ETR  procedure  assumes  data  and  resources  which  are  not 
available  at  the  early  stages  of  system  development.  Thus,  training 
developers  need  guidelines  which  will  aid  them  in  their  assessment  of 
the  effect  of  ETRs  on  system  development. 

Second,  training  development  personnel  need  to  have  access  to  all 
useful  material  as  soon  as  it  becomes  available.  Coordination  of 
schedules  would  help  alleviate  the  problem  of  training  developers  being 
the  last  to  receive  the  newest  pertinent  information.  If  the  training 
developers  are  in  step  with  the  system  developers,  they  will  be  aware 
of  schedule  and  system  changes  and  will  be  able  to  modify  their 
performance  likewise.  This  will  increase  the  efficiency  of  the 
training  developers  by  reducing  the  number  of  iterations  needed  to 
develop  ET  requirements  that  are  consistent  with  the  system. 

To  summarize,  six  problems  concerning  the  collection  of  early 
system  data  were  identified.  It  was  found  that  data  often  are 
inaccurate,  inconsistent  between  sources,  or  missing.  Data  collected 
during  the  early  stages  of  system  development  often  are  not  at  a 
detailed  level  and  can  be  expressed  only  as  functional  data  rather  than 
task  data.  Also  during  these  early  stages,  there  are  often  no  SMEs  to 
whom  one  can  submit  task  lists  for  verification.  A  final  problem  is 
the  lack  of  interaction  and  schedule  coordination  between  system 
developers  and  those  responsible  for  training  development.  This  lack 
of  interaction  can  affect  both  the  final  form  of  the  system  and  the 
efficiency  with  which  the  training  is  developed. 
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EMBEDDED  TRAINING  REQUIREMENTS  SELECTION 


Statement  of  the  Task 


In  the  second  phase  of  the  project,  the  requirement  was  to 
nominate  tasks  from  the  task  lists  generated  in  Phase  One  that  should 
be  considered  for  embedded  training.  These  ETRs  had  to  encompass  tasks 
for  acquisition  training  alone  and  for  both  acquisition  and  sustainment 
training.  To  as  great  an  extent  possible,  task  selection  was  to  be 
made  based  on  the  procedures  outlined  in  Roth  et  al.,  1986. 


Methodology 


$ 

The 

the  ETR 

this  rev 

n' 

A." 

cation. 

all,  the 

that  are  primarily  psychomotor  in  nature.  However,  many  of  the  tasks 
for  ASAS/ENSCE  are  cognitive  in  nature  and  the  procedures  are  not 
applicable  as  written.  The  procedures  for  the  selection  of  ET 
requirements  use  the  classifications  for  objectives  that  rely  heavily 
on  examples  of  psychomotor  tasks  such  as  "start  turbine  engine"  and 
"load  and  fire  howitzer,"  although  the  definitions  do  acknowledge 
decision  skills  and  rule  utilization.  It  is  likely  that  persons  using 
the  guidelines  will  rely  on  the  examples  of  the  use  of  the 
classifications  as  frequently  or  more  so  than  on  the  descriptions  of 
the  classifications.  It  is  true  that  many  tasks  that  are  stated  in 
psychomotor  terras  contain  a  large  cognitive  element.  Unfortunately, 
stating  the  task  in  psychoraotor  terras  often  obscures  the  importance  of 
the  cognitive  tasks  (or  subtasks)  that  are  elements  of  the  overall 
task.  Therefore,  to  make  the  guidance  for  classifying  objectives  more 
useful,  examples  were  developed  that  were  obviously  cognitive  in 
nature.  Second,  the  procedures  for  ETR  nominations  are  based  on 
complexity  and  criticality  ratings  that  require  SME  validation. 

However,  this  validation  was  absent  due  to  the  lack  of  SMEs.  Third, 
the  ancillary  data  associated  with  the  tasks  that  were  available  was 
not  complete  enough  to  allow  for  some  of  the  more  detailed  judgments  to 
be  made.  Fourth,  the  ETR  guidelines  are  designed  for  the  selection  of 
tasks  to  be  sustained  within  the  unit,  rather  than  those  that  are  to  be 
acquired  at  the  institution  or  during  transition  training  at  the  unit. 
In  the  case  of  ASAS/ENSCE,  ET  is  expected  to  support  both  acquisition 
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and  sustainment  training.  Thus  the  guidelines  had  to  be  modified  to 
support  both  acquisition  and  sustainment  training. 

Modification  was  made  by  developing  definitions  and  rating  scales 
appropriate  to  ASAS/ENSCE  for  the  factors  of  criticality,  complexity, 
perishability,  and  feasibility.  These  scales  are  as  follows. 

Criticality  Scale 

High  Criticality  =  3.  Failure  to  perform  this  task  correctly  has 
a  high  probability  of  resulting  in  significant  negative  impact  on 
mission  success  (e.g.,  intelligence  product  inaccurate  or  late  in 
reaching  commander,  unrecoverable  damage  to  operational  database). 

Moderate  Criticality  =  2.  Failure  to  perform  this  task  correctly 
has  some  chance  of  resulting  in  negative  impact  on  mission  success; 
probability  is  low,  impact  is  expected  to  be  slight,  or  error  is 
generally  correctable;  includes  effects  on  efficiency  of  information 
processing  or  dissemination. 

Low  Criticality  =  1.  Failure  to  perform  this  task  correctly  has 
trivial  impact  on  mission  success,  or  is  in  all  cases  correctable  with 
little  delay. 

NOTE:  In  Military  Intelligence  (MI),  the  essence  of  mission 

success  is  that  intelligence  information  is  provided  to  the 
commander  in  a  timely  fashion.  The  elements  leading  up  to 
mission  success  include  acquisition  of  relevant  data, 
accurate  analysis  and  interpretation  of  data,  and  timely 
dissemination  of  the  resulting  intelligence  information  (to 
the  next  stage  of  analysis  or  to  the  command  structure). 
Thus,  a  critical  task  is  one  with  a  potential  to  impact  the 
relevance,  accuracy,  or  timeliness  of  intelligence.  All  of 
these  factors  are  situation-dependent;  i.e.,  the  items  of 
data  which  are  important,  the  factors  influencing  interpre¬ 
tation,  and  the  size  of  the  time  window  for  utilization  of 
intelligence  are  all  variables. 

Because  of  the  lack  of  adequate  information  to  define  the 
situational  factors,  a  worst-case  approach  to  assigning 
criticality  ratings  must  be  used.  For  example,  if  an 
analyst's  failure  to  perform  a  task  correctly  would  result 
in  a  single  item  of  data  being  lost,  it  must  be  assumed 
that  the  lost  data  item  could  be  critical  to  a  specific 
situation.  Similarly,  if  analyst  error  could  cause  a 
delay,  it  must  be  assumed  that  the  delay  could  be  signifi¬ 
cant  in  relation  to  a  specific  time  window.  These 
assumptions  almost  certainly  skew  the  ratings  toward  the 
High  Criticality  end  of  the  scale.  This  is  only 
correctable  given  additional  information  and  SME  support. 
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Complexity  Scale 


The  complexity  scale  was  designed  to  accommodate  the  cognitive 
nature  of  ASAS/ENSCE  tasks.  The  complexity  scale  appears  in  Table  1. 


Perishability 


As  in  the  ETR  procedures  presented  in  Roth,  et  al.,  1986,  perish¬ 
ability  is  directly  based  on  complexity.  It  is  possible  that  retention 
of  the  skill  may  be  independent,  or  at  least  partially  so,  from  com¬ 
plexity,  such  may  be  the  case  when  a  task  requires  many  steps  but  the 
system  prevents  the  soldier  from  deviating  from  those  steps.  But  for 
the  work  discussed  in  this  report,  the  perishability  was  defined  in 
terms  of  complexity  as  in  Roth,  et  al.  (1986).  Thus,  the  following 
rule  was  used: 


If  Complexity  =  1,  then  Perishability  =  1 
If  Complexity  =  2  or  3,  then  Perishability  =  2 
If  Complexity  =  A,  then  Perishability  =  3 


Note  that  the  highest  level  of  Perishability  is  3,  which  is  correlated 
with  the  highest  level  of  Complexity  which  is  4. 


Feasibility 


The  current  ETR  report  (Roth,  et  al.,  1986)  presents  a  flowchart 
of  "implementation"  decisions  which  essentially  rate  the  feasibility  of 
presenting  training  for  each  objective.  For  most  of  the  decisions 
presented,  appropriate  data  were  not  available.  Thus,  this  flowchart 
was  replaced  with  a  much  simpler  set  of  Yes-or-No  questions,  as 
follows : 


1.  Can  the  equipment  provide  the  stimuli  needed  to  train  the 
task? 


2.  Will  the  provision  of  embedded  training  be  free  from  hazard 
to  personnel  and  equipment? 


Certainty  Ratings 


Each  task  identified  in  the  task  analysis  was  rated  on  the  above 
factors.  However,  the  ratings  for  criticality  and  complexity  were  also 
assigned  a  "certainty  rating"  by  the  project  staff  members  examining 
the  data.  The  "certainty  rating"  was  assigned  in  order  to  reflect  the 
fact  that  no  (or  very  little)  SME  input  was  available  to  guide  the 
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Complexity  Scale  for  ASAS/ENSCE 


Code 

Title 

Description 

Examples 

1 

Basic  Cognitive/ 
Behavior  Skill 

Basic  skills  required  to 
operate  equipment  or 
perform  cognitive  task 

Read  at  8th  grade 
level;  recall  pass¬ 
word;  press  Enter  key 

2 

Rule  or  Concept 

Uti lization 

Classification  or  decision¬ 
making  tasks  based  on 
applying  concepts  or  rules 
to  available  information  in 
given  situations 

Determine  message 
routing;  identify 
enemy  units;  identify 
high-value  targets 

Specific  procedures  based  Edit  message  text; 
on  basic  skills  and  rule/  modify  situation  map; 
concept  utilization,  and  set  alert  criteria 
including  flexible  responses 
to  contingencies  and 
variations  in  conditions 


4  Complex  Inte-  Coordinated  task  performance  Develop  and  apply 

grative  Cognitive/  requiring  manipulation  of  hypotheses  about 
Behavioral  large  quantities  of  data,  enemy  plans;  correlate 

Performance  contingency-based  applica-  information  received 

tion  of  rules  in  dynamic  from  multiple  sources; 

situations,  or  rapid  inte-  visualize  and  mentally 
gration  and  synthesis  of  rotate  possible 
sensory  information  avenues  of  approach 


3  Sequential 
Cognitive/ 
Behavioral  Skill 


decisions  made  concerning  Cask  complexity  and  criticality.  The 
"certainty  scale"  values  were  assigned  as  follows: 

1  =  Very  uncertain  of  assigned  rating;  rating  is  a  guess;  SME 

input  essential  to  substantiate. 

2  =  Moderately  certain  of  rating;  educated  guess;  SME  input 

desirable  to  validate. 

3  =  Very  certain  of  assigned  rating;  based  on  knowledge  of 

soldier's  mission  and  type  of  task;  SME  review  preferred 
but  not  urgently  required. 

After  values  for  all  of  the  factors  described  had  been  assigned  to 
a  task,  these  rating  assignments  were  examined  to  determine  whether  the 
task  should  be  selected  for  ET. 

Two  algorithms  for  ET  nomination  were  developed;  one  for 
acquisition  tasks  and  the  other  for  tasks  requiring  sustainment.  The 
following  rules  were  used  for  ET  nomination: 

1.  Answering  "No"  to  either  of  the  feasibility  questions  would 
remove  it  from  consideration  as  an  ETR. 

2.  Nominate  objective  for  acquisition  training  using  ET, 
unless  both  Complexity  and  Criticality  equal  one  (1) 
(Complexity  scale  from  4  to  1;  Criticality  scale  from  3  to 
1). 

3.  Objective  will  be  selected  for  sustainment  training  if 
first  nominated  by  rule  Number  2  above  and  Perishability 
does  not  equal  one  (1). 

The  rule  for  the  nomination  of  tasks  for  sustainment  training  is 
comparable  to  the  algorithm  for  task  selection  that  appears  in  Roth,  et 
al.,  1986.  The  second  step,  that  initially  determines  whether  a  task 
is  to  be  selected  for  ET  at  all,  is  based  on  the  task's  criticality  and 
complexity.  This  rule  reflects  the  notion  that  critical  tasks, 
regardless  of  complexity,  and  complex  tasks,  regardless  of  criticality, 
should  be  initially  taught.  The  decision  that  it  is  not  feasible  to 
train  the  task  using  ET  (rule  1  above)  does  not  rule  out  the 
possibility  that  the  task  will  need  to  be  trained,  only  that  the  task 
is  not  suitable  for  ET.  The  selection  of  training  media  for  tasks 
unsuitable  for  ET  was  not  considered  within  the  context  of  this 
project . 

It  should  be  noted  that  although  only  acquisition  and  sustainment 
training  are  explicitly  addressed  in  the  algorithm  above,  transition 
training  may  also  need  to  be  considered.  In  many  respects,  the  needs 
of  a  person  undergoing  transition  training  should  be  very  similar  to 
someone  receiving  acquisition  training.  Therefore  it  seems  reasonable 
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to  assume  that  rule  2  above  should  hold  for  either  acquisition  or 
transition  training. 


Lessons  Learned 


During  the  course  of  preparing  ET  requirements  for  the  ASAS/ENSCE 
system,  two  major  lessons  learned  and  two  questions  for  future  research 
were  identified.  The  lessons  learned  will  be  addressed  initially  in 
this  subsection  followed  by  a  discussion  of  issues  for  further  ET 
research. 


Guidelines  for  Emerging  Systems 


One  of  the  lessons  identified  was  that  the  existing  guidelines 
(Roth,  et  al.,  1986)  for  the  nomination  of  ET  requirements  are  not 
comprehensive  enough  for  application  to  all  systems  for  which  ET  may  be 
appropriate.  Specifically,  the  guidelines,  as  currently  written,  are 
not  applicable  to  systems  of  the  following  types: 


1. 


Emerging  systems  for  which  there  is  little  or  no  available 
SME  support. 


2. 


Emerging  systems  for  which  there  is  little  or  no  informa¬ 
tion  available  on  the  ease  of  ET  implementation. 


3. 


Systems  that  contain  many  tasks  which  are  cognitive  in 
nature. 


4.  Systems  that  will  use  ET  for  both  sustainment  and  acquisi¬ 
tion  training. 


The  ASAS/ENSCE  system  is  representative  of  all  four  of  the  above 
mentioned  system  types.  Thus,  it  was  necessary  to  modify  the  ETR 
guidelines  to  better  accommodate  the  system  under  examination.  These 
modifications  were  described  in  the  previous  subsection. 


Certainty  Factors 


The  second  lesson  learned  came  about  as  a  consequence  of  the  lack 
of  SME  support  available  for  the  ASAS/ENSCE  system.  Since  there  was 
minimal  SME  input  into  the  task  descriptions  and  their  ratings  for 
complexity  and  criticality,  a  method  had  to  be  devised  to  indicate 
task-specific  needs  for  SME  review.  The  ET  certainty  factors  were 


assigned  to  the  judgments  made  by  project  staff  for  task  complexity  and 
criticality  in  order  to  indicate  to  future  training  developers  the  need 
for  SME  clarification  for  the  ETR  decisions  made. 
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Further  Research 


The  lack  of  SME  support  for  the  selection  of  ET  requirements  and 
the  means  used  for  compensating  for  this  problem  raise  two  issues  which 
need  to  be  addressed  by  further  research.  First  of  all,  the  utility  of 
the  certainty  factors  assigned  to  task  complexity  and  criticality  must 
be  ascertained.  It  is  possible  that  these  factors  may  be  useful  in 
actual  ET  requirement  selection.  However,  research  should  be  conducted 
to  delineate  the  role  of  these  ratings  for  decision  making. 

A  second  question  that  should  be  examined  is  the  actual  signifi¬ 
cance  of  SME  input  for  the  ET  requirements  selection  process.  This 
issue  is  of  vital  importance  in  the  case  of  emerging  systems.  For 
emerging  systems  there  are  no  persons  who  are  experts  at  the  level 
necessary  to  inform  training  recommendations  and  decisions.  In  other 
words,  there  is  no  one  who  is  truly  knowledgeable  in  the  use  of  the 
system  as  it  will  be  in  the  field.  Nevertheless,  recommendations  and 
decisions  concerning  training  development  must  be  made.  These 
recommendations  and  decisions  will  rest  with  persons  who  will  not  have 
access  to  the  type  of  information  specified  by  the  ET  guidelines.  The 
question  is,  then,  how  accurate  are  the  ET  requirements  decisions  made 
by  training  developers  without  SME  support  when  compared  to  those  that 
are  made  based  on  information  supplied  by  SMEs?  An  answer  to  this 
question  will  aid  in  the  determination  of  the  extent  one  may  rely  on 
training  recommendations  and  decisions  made  for  emerging  systems, 
especially  during  their  early  phases  of  development. 
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EMBEDDED  TRAINING  SYSTEM  DESIGN  AND  INTEGRATION 


Statement  of  the  Task 


The  third  phase  of  the  project  was  the  development  of  ET  design 
and  system  integration  concepts  for  ASAS/ENSCE.  It  was  expected  that 
the  results  of  this  third  phase  would  offer  input  to  the  training 
personnel  involved  in  decision-making  processes  concerning  ET  for 
ASAS/ENSCE.  As  such,  the  results  of  this  phase  were  designed  to  be  as 
detailed  as  possible. 

The  ET  design  concept  needed  to  include  information  as  to  what 
would  be  taught,  how,  and  when  during  the  learning  process.  The 
integration  concept  focused  on  how  ET,  as  a  component  of  a  training 
system,  should  be  integrated  with  the  other  components  of  the  system. 
The  integration  concept  also  detailed  the  way  in  which  ET  could  be 
integrated  with  the  operational  system. 


Methodology  For  Development  of  ASAS  ET  Design 


Constraints  on  ET  Design  Concept  Methodology 

It  was  determined  that  there  were  three  major  factors  that 
constrained  the  design  of  ET  for  ASAS.  These  were  the  training  needs 
of  JTFPO,  the  quality  of  the  available  task  data,  and  the  availability 
of  design  decision  support  by  SMEs  knowledgeable  about  various  aspects 
of  the  system.  Each  of  these  factors  shaped  the  final  product  in 
significant  ways. 

Training  Needs.  JTFPO  defined  the  general  characteristics  of  ASAS 
training  that  the  training  design  had  to  address.  The  input  that  was 
received  concerning  JTFPO's  training  needs  indicated  that  ET  would  be 
called  upon  to  support  the  following: 

1.  Institutional  use  on  operational  equipment,  simulators,  and 
personal  computers  (Note:  it  was  clear  to  JTFPO  that  the 
latter  two  configurations  are  not  ET  configurations  in  the 
strictest  sense). 

2.  Unit  use  on  operational  equipment,  possibly  with  a 
"strap-on"  device  to  deliver  software  that  will  "stimulate" 
the  operational  software  and  thus  integrate  it  with  the 
training  database. 
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3.  Training  of  individuals  on  non-collective  tasks. 

4.  Training  of  individuals  on  collective  tasks  in  both  a  mode 
in  which  input  from  other  individuals  is  simulated  by  the 
system  and  in  a  mode  in  which  said  input  is  actually 
supplied  by  other  individuals  through  a  Local  Area  Network 
(LAN). 

5.  Training  for  acquisition,  transition,  and  sustainment  of 
job  skills. 

6.  Training  that  is  presented  both  as  tutorials  with  requisite 
practice,  feedback,  and  evaluation,  and  presented  in 
scenario  exercises  in  which  feedback  and  evaluation  will  be 
deferred. 

Task  Data.  Another  major  constraint  on  the  development  of  the 
ASAS  ET  design  concept  was  the  quality  and  veracity  of  the  task  data  on 
which  the  concept  is  based.  Due  to  the  fact  that  the  system  under 
study  is  an  emerging  system,  any  task  list  developed  prior  to  the 
actual  fielding  of  the  system  must  be  accepted  as  an  approximation  to  a 
particular  stage  in  system  development.  As  an  emerging  system, 
decisions  are  still  being  made  as  to  the  capabilities  that  ASAS  will 
possess.  Also,  for  an  emerging  system  such  as  ASAS,  a  task  list  may  be 
accurate  in  the  elements  that  it  contains,  but  lacking  certain  pieces. 
This  problem  occurs  because  many  aspects  of  a  soldier's  job  utilizing 
the  system  will  not  be  defined  until  after  fielding  of  the  system  and 
its  integration  within  the  unit.  For  example,  as  a  system  for  handling 
data,  ASAS  may  have  the  capability  for  the  production  of  paper  copies 
of  data.  At  this  point  in  the  development  of  the  system,  however, 
policy  has  yet  to  be  set  that  could  guide  a  soldier's  decision  to 
produce  paper  copies  of  information. 

The  development  of  operator  tasks  for  any  system  requires  training 
developers  to  have  access  to  current  documentation  and  SMEs.  For  an 
emerging  system,  this  approach  contains  two  major  pitfalls.  First,  as 
the  system  concept  changes,  so  does  the  documentation  reflecting  the 
state  of  the  system.  Thus,  training  developers  must  be  aware  of  the 
currency  of  their  documentation  and  be  prepared  to  integrate  infor¬ 
mation  from  new  sources  into  their  analyses  as  it  becomes  available. 
Training  developers  also  must  establish  an  effective  method  for 
updating  the  documentation  as  the  operational  system  evolves.  Second, 
for  new  systems,  quite  often  there  are  no  SMEs  with  the  appropriate 
background  knowledge  to  aid  in  determining  the  veracity  of  tasks,  the 
task  components,  task  complexity,  and  task  criticality.  The  lack  of 
SME  "expertness"  is  especially  noticeable  when  the  tasks  in  question 
contain  a  large  cognitive  element,  since  documentation  rarely  indicates 
the  detailed  task  substeps  and  the  knowledge  used  for  the  performance 
of  these  types  of  tasks.  It  is  for  these  reasons  that  training 
analysis  and  design  must  be  performed  in  an  iterative  fashion,  so  that 
training  developers  and  decision  makers  can  estimate  system 
requirements  for  training  based  on  the  most  valid  analysis. 
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In  the  case  of  ASAS,  the  available  documentation  allowed  for  the 
determination  of  procedural  steps  for  many  tasks,  although  not  to  a 
detailed  psychomotor  level.  However,  the  details  of  tasks  that  were 
derived  from  documentation  have  changed  due  to  subsequent  modifications 
in  the  system.  The  substeps  that  comprise  identified  cognitive  tasks 
could  not  be  determined  and  had  to  be  assumed. 

These  imperfect  data  were  subjected  to  a  procedure  to  determine  ET 
requirements  for  ASAS.  The  procedure  used  is  detailed  in  Evans,  et 
al.  (1987b)  and  in  the  previous  section  of  this  document. 

SME  Availability.  The  final  constrain,  on  the  design  of  an  ET 
component  is  the  availability  of  SME  input  to  aid  in  decisions 
concerning  the  contents  of  such  a  component  for  a  system.  By 
"contents"  it  is  meant  as  such  things  as  the  actual  data  items  for  the 
training  component  tutorials  and  scenarios,  specific  wording  of  all 
feedback  and  test  items,  and  detailed  information  concerning  student 
performance  behaviors  to  be  recorded.  As  stated  previously,  such 
support  was  not  available  for  the  ASAS  ET  design  effort. 


Concept  Development 


There  are  three  approaches  that  one  may  select  from  with  regard  to 
ET  component  design  for  an  operational  system  in  an  early  stage  of 
development.  The  first  option  is  to  wait  until  the  design  of  the 
operational  system  has  stabilized  prior  to  initiation  of  ET  component 
design.  This  approach  is  economical  in  that  fewer  iterations  of  the 
design  process  are  required  prior  to  the  development  of  one  that  is 
appropriate  to  the  operational  system.  The  drawback  of  this  approach 
is  that  the  needs  of  the  training  system  are  considered  subsequent  to 
finalization  of  the  operational  system,  a  situation  which  could  lead  to 
a  lack  of  planning  for  the  incorporation  of  ET  into  the  system. 


A  second  option  is  to  develop  a  general  ET  design  concept  at  an 
early  stage  in  the  development  of  the  operational  system.  This  design 
would  address  issues  for  consideration  as  the  operational  system  is 
developed.  The  design  would  be  a  framework  in  which  to  place  detail  as 
it  becomes  available. 


The  primary  drawback  of  this  second  approach  is  that  a  general 
design  concept  may  not  include  enough  detail  until  its  final  manifes¬ 
tation  to  offer  input  into  the  development  of  ET  for  the  system.  In 
this  option,  constraints  resulting  from  the  design  of  the  operational 
system  drive  the  design  of  the  training,  rather  than  training  factors 
impacting  the  operational  system  design  in  any  meaningful  way. 


A  third  approach  exists  that  is  more  resource-consuming  than  the 
first  two  systems,  but  can  supply  input  to  the  operational  system 
developers  so  that  tney  can  integrate  ET  design  requirements  into  the 
operational  system.  This  third  option  requires  the  development  of 
several  detailed  ET  design  concepts.  This  ET  concept  development  would 


occur  at  as  early  a  stage  in  operational  system  design  as  possible. 
Each  ET  design  concept  would  be  based  on  all  known  data  concerning  the 
operational  system,  as  well  as  a  -et  of  well-stated  assumptions.  The 
variance  between  designs  would  b^  in  their  assumptions.  As  details 
about  the  operational  system  become  known,  the  designs  may  be  revised 
until  the  one  that  most  resembles  the  appropriate  training  system  can 
be  determined. 

The  strength  of  this  third  option  is  that  although  the  training 
design  is  modified  in  response  to  the  changes  in  the  operational 
system,  the  designers  of  the  operational  system  are  constantly  kept 
aware  of  the  training  design  requirements  that  the  operational  system 
must  accommodate.  It  is  through  this  interplay  between  the  needs 
expressed  by  the  training  developers  early  in  the  system  design  cycle 
and  the  parameters  delineated  by  the  system  developers  that  ET  can 
become  truly  a  part  of  the  operational  system. 


Documentation  Review 


In  order  to  develop  the  ASAS  ET  design  concept,  the  utility  for 
application  of  two  documents  dealing  with  training  design  for  computer- 
based  systems  was  Investigated.  One  document  was  the  procedure 
developed  specifically  for  the  production  of  ET  design  components 
(Fitzpatrick,  et  al.,  1987).  The  other  document.  An  Enhanced  Instruc¬ 
tional  Design  Process  for  Developing  Interactive  Courseware  (Marco,  et 
al.,  1986),  was  focused  on  computer-based  training,  in  general,  and  not 
specifically  on  ET.  Both  documents  were  examined  to  determine  which 
aspects  of  the  documents  were  applicable  to  the  development  of  the  ET 
concept  for  ASAS. 


ET  Design  Guidelines.  Since  the  task  at  hand  was  to  develop  an  ET 
design  concept  for  ASAS,  the  first  document  that  was  examined  speaks 
directly  to  the  design  of  ET,  the  existing  guidelines  for  the  design  of 
an  ET  component  (Fitzpatrick,  et  al.,  1987).  It  is  focused  on  using 
task  objective  data  to  produce  a  component  for  ET  that  contains  the 
training  database  and  details  for  its  implementation.  It  requires  the 
development  of  a  detailed  method  for  training  each  ET  objective. 
Therefore,  all  procedures  specified  by  the  report  are  to  be  performed 
on  each  ET  objective.  The  procedures  are  divided  into  six  phases. 

Upon  examination  of  these  phases,  it  was  decided  that  although  certain 
concepts  from  the  first  two  phases  were  applicable  to  the  data  for 
ASAS,  the  later  phases  required  input  and  produced  output  at  a  level  of 
detail  that  currently  cannot  be  supported  by  the  ASAS  data. 


The  first  step  in  designing  an  ET  component  is  to  review  the  tasks 
selected  for  ET  and  convert  these  tasks  into  training  objectives.  The 
first  portion  of  this  phase  requires  the  development  of  a  database 
containing  ET  requirements  to  which  training  priorities  have  been 
assigned.  This  task  had  been  accomplished  during  Phase  Two  of  the 
project  (selection  of  ETRs  for  acquisition  and  sustainment).  The 
second  part  of  this  step  is  to  develop  training  objectives  based  upon 


m 


i 


ft 


* 


& 


I 


v>; 


s 

y. 


ft 


vwjrxmrir*. yvy  w.'awwm  wwyy^gi 


Che  collected  task  data.  It  was  determined  that  due  to  the  lack  of 
detailed  task  information,  enabling  skills  and  knowledges  for 
objectives  could  not  be  stated.  However,  it  was  felt  that  the  tasks 
themselves  could  serve  as  objectives. 


In  the  second  step  of  ET  component  design,  several  variables 
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affecting 

design  are  identified.  These  include  the 

Ko 

1. 

Or.e  or  more  training  approaches. 

2. 

Job-related  stimuli. 

3. 

Fidelity  needs. 

A 

4. 

Stimulus  categories. 

5. 

Performance  measures  for  each  objective 

6. 

Feedback  events. 

ft 
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7. 

Recording  everts. 

8. 

Stimuli  implementation  strategies. 

For  ASAS,  it  was  determined  that  some  of  these  concepts  could  be 
utilized  in  the  design,  but  possibly  not  in  the  form  described  in  the 
ET  design  document.  For  example,  a  training  concept  could  be  devised 
partially  based  on  the  answers  to  the  questions  used  to  identify  an 
approach.  However,  there  was  not  enough  information  to  identify  any 
job-related  stimuli  for  tasks  beyond  the  equipment  in  use  for  the 
performance  of  the  task.  Nor  could  specific  statements  concerning 
performance  measures,  feedback  events,  or  recording  events  be  made 
without  input  from  SMEs  unless  many  assumptions  were  made.  For 
emerging  systems,  these  assumptions  should  be  made,  subject  to  future 
modification,  and  documented.  It  was  decided  that  the  concept  of 
fidelity,  as  stated  in  the  Fitzpatrick,  et  al.  (1987)  document,  could 
be  amended  and  applied  to  the  task  data  in  that  the  basis  for  fidelity 
decisions  was  stated  as  being  task  criticality.  These  types  of  data, 
although  not  verified,  were  available.  The  concept  of  fidelity  was 
expanded,  given  input  from  the  second  document  that  was  examined.  ET 
requirements  could  also  be  categorized  at  a  general  level  in  terras  of 
the  types  of  stimuli  required.  Since  both  fidelity  requirements  and 
necessary  stimuli  could  be  identified,  task-specific  statements 
concerning  stimulus  implementation  could  be  made. 


It  was  also  felt  that  guidance  for  making  some  decisions  required 
by  the  ET  design  document  was  lacking.  For  example,  the  procedure  for 
selecting  feedback  presents  definitions  for  different  types  of  feedback 
without  mentioning  how  to  select  between  these  different  types. 


Marco,  et  al.,  Document.  The  second  document  that  was  found 


relevant  for  this  effort  (Marco,  et  al.,  1986)  gives  guidance  in  the 


19 


-A 


development  of  training  design  concepts  for  interactive  computer-based 
courseware,  as  would  be  developed  for  an  ET  application.  Although  this 
document  does  not  specifically  address  the  issue  of  design  for  ET,  it 
was  felt  that  many  of  the  concepts  and  the  approach  described  by  Marco 
and  her  colleagues  were  extremely  applicable  to  the  development  of  an 
ET  design  concept  to  be  based  on  partial  data. 

Marco's  approach  developed  out  of  need  to  produce  Computer-Based 
Training  (CBT)  for  the  Model  Training  Program  for  Reserve  Component 
Units  (MTP-RC).  She  and  her  colleagues  identified  weaknesses  in 
existing  guidelines  for  producing  CBT.  These  weaknesses  included  such 
problems  as  an  inefficient  approach  that  iterates  the  same  design  steps 
over  many  lessons,  little  guidance  for  making  design  decisions,  and  no 
conceptual  tools  given  for  designing  non-linear  media.  Thus,  the 
objectives  of  their  document  were  threefold.  First,  Marco  and  her  team 
wanted  to  define  a  way  in  which  similar  tasks  could  be  identified  and 
grouped  so  as  to  be  able  to  create  instructional  designs  that  can  be 
applied  to  lessons  in  which  similar  skills  are  taught.  Second,  the 
team  aimed  at  developing  guidance  for  making  design  decisions 
concerning  various  training  variables,  including  decisions  on  mastery 
training,  feedback,  and  learner  control  of  lesson  presentation.  The 
third  objective  of  the  Marco  team  was  to  provide  design  tools  and 
techniques  that  would  separate  the  lesson  content  from  program  logic. 

It  is  the  first  two  objectives  of  the  Marco  group  which  were  addressed 
in  the  ASAS  situation.  The  third  objective  was  of  interest,  but  it  was 
felt  that  within  the  context  of  this  project,  the  information  and 
resources  necessary  to  develop  tools  for  program  logic  were  not 
available  and  thus  this  objective  was  not  pursued. 

The  first  step  in  the  process  developed  by  Marco,  et  al.  (1986)  is 
the  selection  of  the  task  content  for  the  training.  According  to  the 
Marco  group,  there  are  four  factors  that  affect  content  selection. 

These  are  institution,  audience,  content  variables,  and  resources 
available  for  training  development.  One  must  assess  the  effect  on 
content  selection  by  answering  training-relevant  questions  for  each 
factor  in  turn.  Typical  issues  that  might  be  addressed  include: 

Institution:  Training  location 

Training  frequency 
Mode  of  training  delivery 

Factors  leading  to  the  development  of  training  at 
this  time 

Audience:  Composition  of  the  audience 

Current  level  of  expertise  of  the  audience 
Length  of  time  at  current  level  of  expertise 
Level  of  motivation  for  learning 
Level  of  familiarity  with  the  training  medium 

Content:  Task  criticality 

Task  stability  or  perishability 
Frequency  of  need  to  perform  task 
Task  transferability 
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Resources:  Time  available  for  training  development 

Time  availability  for  training 
Budgetary  constraints 

SMEs  available  for  training  development  and  their 
cost 

In  the  case  of  ASAS,  content  for  training  was  developed  by  the 
application  of  procedures  for  the  selection  of  ET  requirements 
described  in  Evans,  et  al.,  (1987b)  and  in  this  report.  It  is  useful, 
however,  to  examine  the  effect  on  content  selection  for  ASAS  training 
that  occurs  when  one  looks  at  the  issues  identified  by  Marco,  et  al. , 
as  related  to  content  decisions  for  ASAS  ET.  For  example,  it  is  known 
that  ASAS  ET  is  to  be  delivered  at  both  the  institution  and  the  unit, 
thus  tasks  for  either  or  both  acquisition  and  sustainment  training  must 
be  identified.  The  frequency  of  delivery  will  vary  given  that  the 
training  will  serve  more  than  one  function.  The  audience  for  computer- 
based  training  will  consist  of  military  intelligence  analysts  of 
various  MOSs  and  skill  levels.  These  students  will  already  be  versed 
in  MI  analysis  using  manual  procedures,  but  they  may  not  be  at  all 
familiar  with  computer  systems  such  as  ASAS.  The  audience  will  also 
consist  of  maintenance  personnel  and  others  performing  non-analyst 
duties.  Since  training  will  be  for  several  levels  and  types  of 
personnel,  the  training  should  reflect  the  differences  in  MOSs  and 
skill  levels.  The  initial  training  also  should  be  airapd  at  computer 
novices.  On  the  other  hand,  detailed  knowledge  concerning  content 
issues  is  uncertain;  the  frequency  of  task  performance  is  unknown,  as 
is  skill  and  knowledge  transferability  from  the  manual  mode  to  the 
automated;  and  knowledge  concerning  task  criticality,  complexity,  and 
perishability  is  shaky.  Thus  it  is  necessary  to  select  tasks  for 
training  rather  than  to  reject  them  until  clarification  on  these  issues 
can  be  obtained.  With  regard  to  the  final  factor,  availability  of 
resources  for  training  development,  resources  will  constrain  the  tasks 
eventually  selected  for  training  development  and  implementation,  but 
the  audience  resources  should  not  impact  upon  the  identification  of 
tasks  requiring  training  nor  the  methods  by  which  these  tasks  should, 
ideally,  be  trained. 

Given  the  knowledge  available  on  these  issues  identified  by  Marco 
and  her  colleagues  as  being  important  for  the  determination  of  training 
content,  the  comparison  of  the  tasks  that  might  be  selected  for 
training  using  the  Marco  strategy  and  those  selected  by  the  modified  ET 
requirements  procedure  used  at  an  earlier  stage  in  this  project,  would 
probably  result  in  the  development  of  very  similar  lists  of  tasks  to  be 
trained.  However,  such  an  overlap  might  not  occur  for  systems  for 
which  the  current  ET  requirements  guidelines  are  applicable.  The 
correlation  between  findings  using  the  Marco  procedure,  the  modified  ET 
procedure,  and  the  current  ET  requirements  guidelines  should  be 
investigated  empirically  to  determine  if  the  results  of  early 
procedures  are  acceptably  predictive  of  later  system  training  needs. 
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After  determining  the  content  of  training,  Marco  and  her 
colleagues  perform  a  training  task  analysis  that  results  in  training 
objectives  and  their  enabling  skills  and  knowledges.  They  identify 
commonalities  between  skills  and  knowledges  and  cluster  the  skills  and 
knowledges  into  groups  that  represent  the  selected  dimensions. 

As  mentioned  previously,  the  tasks  selected  for  ET  were  used  as 
the  training  objectives.  However,  after  examining  the  ASAS  ET  require¬ 
ments,  at  least  three  ways  were  identified  in  which  to  group  tasks. 

One  way  focused  on  the  commonalities  between  soldier-machine  interface 
tasks  that  appear  for  all  functional  areas,  to  greater  and  lesser 
extent.  This  approach  produced  the  following  groups:  file  management 
tasks,  database  management  tasks,  message  production  tasks  (word 
processing),  and  graphics  production  tasks.  These  task  groups  apnear 
as  both  CU  tasks  and  within  MI  problem  solving  contexts  for  the  func¬ 
tional  areas.  For  example,  all  personnel  perform  some  types  of  data¬ 
base  management  tasks,  such  as  information  retrieval,  but  personnel  in 
some  of  the  functional  areas  will  be  required  to  perform  more  complex 
database  management  tasks  (e.g.,  information  correlation)  than  other 
personnel.  Another  clustering  of  tasks  consisted  of  overt  behavioral 
procedures  versus  cognitive  procedures.  The  third  grouping  of  tasks 
was  the  separation  of  the  CU  tasks,  which  appear  in  all  function- 
specific  contexts,  and  the  functional  area  tasks  (CM,  SS,  AS,  TA ,  SA, 
ELINT,  COMINT).  It  was  fe't  the  best  way  to  handle  both  the 
commonalities  between  task.*  and  the  differences  was  to  discuss  training 
in  terms  of  task  types  and  ET  training  structure  (Tiers).  The  first 
part  would  address  the  training  design  for  the  overall  task  groups 
mentioned  above.  The  second  half  of  the  design  concept  would  focus  on 
task-specific  issues  for  tasks  drawn  from  each  functional  area. 

Marco,  et  al.,  identified  several  variables  on  which  to  base  their 
training  design.  These  variables  are  feedback  level,  learner  control, 
passing  level,  simulation  fidelity,  performance  error  fidelity,  and 
level  of  guidance.  They  also  developed  guidance  for  making  decisions 
for  these  variables  that  could  apply  to  one  or  to  a  group  of  similar 
tasks.  These  decisions  concerning  the  training  variables  were  used  to 
develop  templates  for  training  groups  of  skills  and  knowledges. 

The  variables  identified  by  Marco  and  her  colleagues  were  examined 
and  those  that  seemed  the  most  applicable  to  the  ASAS  situation  were 
selected.  These  variables  were  passing  levels,  learner  control, 
feedback,  and  simulation  fidelity.  The  definitions  for  feedback  and 
simulation  fidelity  were  modified  to  be  more  reflective  of  the  concepts 
as  represented  by  the  ET  design  document.  Marco's  concept  of  level  of 
guidance  also  was  selected  but  renamed  "decision  cueing"  to  better 
reflect  the  actual  process.  Marco's  definitions  for  these  variables 
are  as  follows: 

Passing  Levels:  "  ...  the  description  of  the  acceptable 

performance  level.  It  is  the  measure  of  the 
soldier's  mastery  of  the  criterion  of  the 
instructional  objective."  (p.  21) 
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Learner  Control: 


...  the  degree  to  which  the  learner  or  the 
computer  program  controls  the  learner's  path 
through  the  sequence  of  units,  lessons,  or 
segments  of  instruction.  Learner  control 
also  refers  to  who  or  what  controls  the 
selection  of  which  content  to  learn  within  a 
unit,  lesson,  or  segment  of  instruction." 

(P-  21) 

Feedback:  "...  the  procedure  used  to  tell  a  learner  if 

his  response  to  an  instructional  event, 
usually  a  practice  item  or  test  question,  is 
right  or  wrong."  (p.  20) 

Simulation  Fidelity:  "...  the  degree  of  similarity  between  the 

stimuli  and  actions  taken  during  training 
and  the  stimuli  and  actions  taken  during  the 
actual  performance  of  the  task."  (p.  22) 

Level  of  Guidance:  "...  is  concerned  with  the  amount  of 

(Decision  Cueing)  guidance  in  the  form  of  textual  prompts  and 

visual  cues  that  should  be  included  in  a 
lesson."  (p.  24) 

It  was  felt  that  the  variables  selected  from  the  Marco,  et  al., 
paper  required  augmentation  and  modification  to  make  them  more  specific 
to  ASAS.  In  addition,  two  factors  were  added  to  the  list  discussed  in 
the  paragraph  above.  The  first  of  these  factors  was  "relationship  of 
manual  intelligence  procedures  to  ASAS  procedures."  Since  the  MI 
students  learning  how  to  use  ASAS  will  have  received  previous 
instruction  on  the  performance  of  MI  tasks  in  a  manual  mode,  there 
should  be  some  transfer  from  the  concepts  learned  manually  to  their 
implementation  using  ASAS.  This  relationship  can  be  used  to  promote 
efficient  learning  by  indicating  to  the  student  the  similarities  and 
differences  between  the  manual  performance  of  t  task  and  the  ASAS 
version.  Also,  such  transfer  means  the  ASAS  training  will  not  have  to 
address  the  non-ASAS  aspects  of  the  tasks  in-depth.  For  these  reasons, 
it  was  felt  that  it  would  be  informative  to  address  the  transfer  issue 
for  all  ASAS  tasks  to  which  it  pertains. 

The  second  variable  that  was  added  to  the  design  variables  list 
was  a  factor  indicating  the  need  for  demonstrations  of  procedures  that 
are  performed  using  ASAS.  These  demonstrations  would  be  dynamic  and 
would  simulate  operator  behavior.  It  was  felt  that  this  factor  would 
be  useful  for  the  determination  of  the  training  approach  for  both 
individual  tasks  and  for  groups  of  common  tasks. 


After  the  examination  of  the  documentation,  two  lists  of  variables 
were  arrived  at  for  inclusion  in  the  design  concept.  One  list  was 
selected  for  use  in  the  general  training  design  concept  and  the  other 
was  selected  for  the  task-specific  comments.  The  difference  between 
the  two  lists  is  that  the  variable  of  learner  control  is  omitted  from 
the  task-specific  list.  This  omission  is  due  to  the  inability  of 
arriving  at  any  statements  concerning  learner  control  that  did  not 
apply  to  all  tasks.  The  variables  are  as  follows: 


General 

Passing  Levels 

Relationship  of  Manual 

Intelligence  Procedures  to  ASAS 
Procedures 

Procedure  Demonstrations 
Learner  Control 
Feedback 
Decision  Cueing 
Simulation  Fidelity 

These  variables  are  defined  as  follows. 


Task-Specific 

Passing  Levels 

Relationship  of  Manual 

Intelligence  Procedures  to 
ASAS  Procedures 

Procedure  Demonstrations 

Feedback 

Decision  Cueing 

Simulation  Fidelity 


Passing  Levels.  The  acceptable  performance  level  prescribed  by 
TRADOC  to  be  achieved  by  the  individual  trainee. 

Passing  levels  include  any  of  the  following: 

1.  Complete  Mastery  -  All  tasks  are  performed  without  error. 
Complete  mastery  is  recommended  for  tasks  and  skills  which 
are  identified  as  critical,  or  if  errors  could  result  in 
injury,  loss  of  life,  or  damage  to  equipment.  Complete 
mastery  also  applies  to  subtasks  that  are  prerequisite  to  a 
critical  task.  Complete  mastery  requires  not  less  than  a 
100  percent  passing  level. 

2.  Partial  Mastery  -  Trainee  performance  is  required  to  be  at 
a  high  level,  but  complete  mastery  of  tasks  and  subtasks 
(non-critical  only)  is  not  required.  Partial  mastery  is 
appropriate  during  practice  and  instructional  setments  of  the 
training  period.  Percentage  passing  levels  (i.e.,  80 
percent)  will  be  determined  by  TRADOC. 


3.  No  Mastery  (familiarization)  -  Individual  trainee  perform¬ 
ance  is  not  measured.  No  mastery  is  intended  to  provide 
the  trainee  with  general  knowledge  and  information  only. 


Relationship  of  Manual  Intelligence  Procedures  to  ASAS  Pro¬ 
cedures  .  Acquisition  training  at  institutional  and  unit  levels  will 
consist  of  initial  learning  of  manual  techniques  for  intelligence 
gathering  and  processing.  This  will  be  followed  by  training  to  use 
ASAS/ENSCE  tools  to  perform  these  same  tasks.  Numerous  relationships 
exist  between  present  manual  procedures  and  those  envisioned  to  be 
performed  by  ASAS/ENSCE.  For  example,  ASAS/ENSCE  graphic  capabilities 
provide  an  automated  means  of  performing  those  tasks  that  must  be  done 
manually  with  a  map  and  paper  overlays.  These  relationships  must  be 
pointed  out  because  much  of  the  skills  and  knowledge  associated  with 
map  reading  will  be  taught  during  student  manual  procedure  training  and 
need  not  be  addressed  in  detail  during  ASAS/ENSCE  ET. 


Procedure  Demonstrations.  Procedure  demonstrations  where  the 
training  software  simulates  correct  operator  interactions  with  the 
system  would  be  particularly  appropriate  for  simple  soldier-machine 
interface  activities  such  as  system  initialization,  message  access, 
template  completions,  etc.  Higher  level  interactions  such  as  overlay 
generation,  mission  planning,  file  modifications,  etc.,  are  also 
appropriate  for  demonstrations  of  proper  procedure. 


Learner  Control.  Learner  control  refers  to  the  amount  and  type  of 
control  that  a  student  may  have  over  his  interaction  with  a  training 
system.  This  control  can  be  either  in  the  sequencing  of  training  or  in 
the  initiating  or  terminating  of  training  or  both.  He  may  also  be  allowed 
to  choose  a  difficulty  level. 


Feedback.  Feedback  involves  providing  information  to  the  trainee 
about  his  performance  on  a  learning  activity,  such  as  a  practice  item 
or  test  question.  Two  primary  aspects  to  be  considered  when  specifying 
feedback  requirements  for  training  are:  1)  immediacy  of  the  feedback, 
and  2)  comprehensiveness  of  feedback  given.  A  third  aspect  is  feedback 
adaptivity. 


Decision  Cueing.  MI  gathering  and  processing  via  manual  means  is 
assumed  known.  However,  with  the  arrival  of  ASAS/ENSCE,  it  cannot  be 
assumed  that  soldiers  in  the  MI  community  are  computer  literate  and 
possess  adequate  typing  skills  to  function  reasonably  well  within  the 
ASAS/ENSCE  environment.  Therefore,  it  must  be  assumed  the  trainees' 
system  knowledge  and  experience  is  nil. 


Under  these  conditions,  the  level  of  system-based  guidance 
(cueing)  must  be  high  for  at  least  the  initial  stages  of  the  training 
course.  As  the  trainee  advances  through  the  course,  knowledge  and 
experience,  hopefully,  increases.  Thus,  it  can  be  assumed  the  trainee 
will  become  less  and  less  dependent  upon  cues.  Of  course,  cues  or 
prompts  should  be  avoided  for  the  purpose  of  quizzing  or  tests. 
Scenarios  should  also  be  free  of  cueing  when  presented  at  or 


near  the  terminal  point  in  the  training  course.  Conversely,  any 
scenarios  presented  during  the  earlier  stages  in  the  course  probably 
should  contain  appropriate  cues. 

Simulation  Fidelity.  Initial  training  might  require  high  fidelity 
simulation,  but  in  less  than  real  time.  High  fidelity  simulation 
creates  an  enhanced  learning  environment  and  the  less  than  real-time 
running  speed  allows  for  transfer  of  course  objectives  to  the  trainee. 
High  fidelity  simulation  should  also  be  considered  for  tasks  and 
subtasks  which  are  identified  as  critical  or  hazardous. 

Theater-specific  scenarios  should  be  of  high  fidelity,  without 
question  or  debate.  Simulations  of  lower  fidelity  are  appropriate  for 
lower  level  tasks  and  sub-tasks  and  also  where  a  long  chain  of 
procedural  steps  contain  routine  or  repetitive  processes. 


Development  of  Additional  Guidance 


In  the  Marco,  et  al.  document  (1986),  guidance  is  provided  for 
making  decisions  concerning  the  application  of  the  variables  discussed 
in  their  paper.  That  guidance  was  used  in  applying  the  specially 
selected  Marco,  et  al.  variables  to  the 

ASAS  tasks.  Additional  guidance  was  developed  for  use  in 

the  application  of  the  variables  generated  specifically  for  ASAS  (these 
variables  were  "relationship  of  manual  intelligence  procedures  to  ASAS 
procedures"  and  "procedure  demonstrations"). 


Analysis  of  Tasks 


Stable  ASAS  tasks  were  then  analyzed  with  respect  to  the  sets  of  variables  and 
guidance  described  above.  A  task  was  defined  as  stable  if  it  appeared 
consistently  in  different  sources  of  documentation  (e.g.,  Functional 
Area  Expert  Forum  and  Critical  Design  Review  documents)  or  an  SME 
provided  verification  of  the  task.  This  definition  eliminated  many  of 
the  operator  tasks  that  were  impacted  by  system-user  interface  design 
changes  (such  as:  menu  choice  selection  and  input  device  used)  that 
occurred  during  the  time  frame  of  this  project.  The  outcome  of  the 
task  analysis  was  a  task-by-task  discussion  of  training  recommendations 
and  a  more  general  set  of  recommendations.  The  general  design  concept 
was  the  distillation  and  summary  of  common  findings  across  all  of  the 
tasks  that  were  examined.  The  discussion  within  both  the  general 
design  concept  for  ASAS  and  the  task-specific  comments  is  in  terms  of 
the  training  needs  of  JTFPO,  identified  task  groupings,  and  the  Tiers 
in  which  tasks  would  be  trained  as  they  pertain  to  the  task(s).  These 
are  as  follows: 

1.  Training  type:  Acquisition  or  sustainment  or  both 

2.  Training  location:  Institution  or  unit  or  both 


26 


uvuwrvi  uv^-vusn  xn  iuiwi»YT\nmfBVVW/  ’VWJCTrirrTT^  it  i.iuto'.wW’  /vw  -j-»  it*  vt  vnr*ir*i -jt  -J  *>  -J.  ">F,rjr^jvrr 


3.  Training  audience:  Individual  or  collective  or  both 

4.  Task  group:  Files  management,  database  manage¬ 

ment,  text  production,  or  graphics 
production;  overt  behavioral 
procedures  or  cognitive  pro¬ 
cedures;  CU  or  functional  areas 

5.  Training  approach:  Tutorial  or  scenario  or  both 

6.  ASAS  release  schedule:  Early  or  later,  and  integration 

Tier  I  Teaches  common  user  tasks,  such  as  word  processing, 

log-or.  and  log-off,  etc.  It  also  presents  an  overview  of 
the  training  program  and  explains  how  to  interface  with 
ET.  An  overview  of  ASAS/ENSCE  is  also  presented. 
Generally  uses  tutorials  as  training  approach. 

Tier  II  Teaches  ASAS/ENSCE  function-specific  tasks  in  an  MI 

context.  Topics  to  be  included  are  symbology,  message 
traffic,  and  security  policies  and  procedures,  as  well 
as  functional  area  tasks.  Generally  uses  tutorials  as 
training  approach. 

Tier  III  Deals  with  mission  training.  Includes  Red/White/Blue 
scenarios  as  well  as  enclave  specific  scenarios. 

Generally  uses  scenarios  as  training  approach.  The 
scenarios  are  both  static  and  dynamic. 

Tier  IV  Presents  complex,  theatre-specific  scenarios.  Uses 
free-play  scenarios  as  training  approach. 


As  mentioned  previously,  the  concept  for  integrating  ASAS/ENSCE  ET 
needed  to  reflect  two  types  of  integration:  first  with  the  other 
components  of  the  training  system  and,  second,  with  the  operational 
system.  The  first  integration  task  required  an  examination  of  ways  in 
which  tasks  similar  to  those  selected  for  ASAS/ENSCE  may  be  trained: 
instructor-led,  hands-on,  computer-based  self-paced  instruction,  etc. 
Tasks  that  were  nominated  for  ET  may  be  presented  using  other  media. 

The  question  is,  "How  much  of  the  training  for  a  task  is  to  be  trained 
using  the  operational  system  (ET),  on  stand-alone  devices  using 
computer-based  instruction  (CBI),  or  instructors  giving  presentations? 

In  order  to  answer  the  question  stated  above,  current  training 
plans  for  ASAS/ENSCE  were  examined.  These  plans  presented  course- 
length  estimates  based  on  Plans  of  Instruction  (POIs)  for  existing  MI 
courses.  These  estimates  were  combined  with  estimates  of  course  size 
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made  by  examining  the  tasks  to  be  trained  and  their  complexity.  These 
estimates  produced  approximations  of  the  numbers  of  lessons  to  be 
taught . 

A  second  approach  was  used,  also.  First,  tasks  were  grouped  as 
described  in  the  previous  subsection.  These  groupings  included: 

1.  Common  user  tasks — file  management,  word  processing, 
situation  display,  database  management,  security,  message 
reception  and  transmission. 

2.  Functional  area  tasks — CM,  TA,  AS,  SA,  SS,  COMINT,  ELINT. 

3.  Cognitive  tasks — decisions,  analyses,  etc. 

4.  Psychomotor  tasks — data  input  using  alphanumeric  keyboard, 
moving  cursor  control  or  device,  etc. 

For  each  identified  category  of  tasks  and  its  subcategories, 
possible  methods  of  instruction  were  identified.  These  methods  of 
instruction  were  examined  in  terms  of  type  of  task,  its  complexity,  and 
whether  the  task(s)  could  be  demonstrated  via  a  computer.  Approaches 
for  training  categories  of  tasks  and  individual  tasks  were  developed. 
These  approaches  included  discussions  of  the  types  of  media  to  be  used 
to  train  types  of  tasks  and  how  the  media  interrelate.  Definitive 
approaches,  however,  could  not  be  determined,  only  suggestions  for 
alternatives  based  on  different  sets  of  assumptions. 

The  procedure  used  for  determining  the  second  type  of  integration 
concept  (ET  and  operational  system  integration)  was  simple.  Various 
draft  documents  addressing  general  ET  requirements  were  examined  in 
light  of  ASAS/ENSCE  needs.  A  market  survey  of  CBT  products  applicable 
to  ASAS/ENSCE  was  also  examined.  The  information  presented  in  the 
accessed  sources  was  utilized  to  define  issues  for  discussion  and  to 
aid  in  the  development  of  alternatives  for  ET-operational  system 
integration . 


Lessons  Learned 


Several  lessons  were  learned  during  the  process  of  developing  the 
ET  design  and  integration  concepts  presented  in  the  third  document 
produced  for  this  project.  These  lessons  also  suggest  areas  for 
further  research  on  the  topic  of  ET  design  development  and  recommen¬ 
dations  for  guidelines  for  design. 


ASAS  ET  Design 


There  were  two  important  lessons  that  resulted  from  the  develop¬ 
ment  of  the  ET  design  for  ASAS/ENSCE.  First  of  all,  it  was  determined 
that  the  existing  ET  guidelines  for  design  development  are  not  ap¬ 
propriate  for  the  type  of  data  that  were  available  for  ASAS.  The 
procedures  set  forth  in  Fitzpatrick,  et  al.  (1987)  are  geared  for 
systems  whose  operational  design  and  policy  issues  concerning 
utilization  have  been  resolved.  This  focus  assumes  that  the  collected 
data  are  stable,  at  a  detailed  behavioral  level,  and  verified.  This 
situation  does  not  exist  for  ASAS/ENSCE.  The  data  collected  for 
ASAS/ENSCE,  although  the  best  available  at  the  time,  are  neither 
stable,  at  the  correct  level  of  detail,  nor  necessarily  correct.  In 
addition,  the  procedures  produce  a  design  concept  that  is  at  a  very 
detailed  level.  During  an  early  ET  design  iteration  for  an  emerging 
system,  one  must  make  assumptions  about  the  system  and  the  training 
requirements  for  that  system  in  order  to  develop  an  ET  design  concept. 
As  information  becomes  available  the  design  must  be  revised. 

The  existing  guidelines  also  assume  that  there  will  be  or  has  been 
SME  input  available  for  various  training  decisions  that  must  be  made, 
such  as  performance  measure  selection.  Prior  to  system  stabilization, 
this  type  of  input  may  not  be  available,  and  was  not  available  for  this 
particular  system.  Again,  one  may  make  assumptions  and  base  the  design 
on  those  assumptions,  with  the  understanding  that  the  design  will 
require  future  modification. 

To  deal  with  these  problems  for  production  of  an  ET  design  compo¬ 
nent,  a  concept  to  ET  system  design  was  developed  based  on  known  infor¬ 
mation  and  certain  stated  assumptions.  This  design  indicated  issues  of 
importance  for  training  and  recommendations  for  consideration  during 
the  development  of  ET  for  ASAS/ENSCE  tasks.  It  is  foreseen  that  at 
some  later  point  in  time  a  revised  design  will  be  developed. 

During  the  examination  of  the  ET  design  guidelines,  a  deficiency 
was  identified  in  those  procedures  that  render  them  difficult  to  use. 
The  problem  is  that  certain  procedures  define  the  elements  in  question 
but  do  not  explicitly  give  guidance  for  making  decisions  concerning  the 
elements.  A  case  in  point  is  the  discussion  on  the  selection  of  the 
type  of  feedback  to  use  in  training  an  objective.  Several  types  of 
feedback  are  defined,  but  no  rules  for  selection  are  supplied.  To 
overcome  this  problem,  a  document  was  used  that,  although  not  ET 
specific,  offered  the  type  of  guidance  that  was  sought.  This  document, 
An  Enhanced  Instructional  Design  Process  for  Developing  Interactive 
Courseware  (Marco,  et  al.,  1986),  also  supplied  a  framework  with  which 
to  approach  the  problem  of  developing  an  ET  design  concept  for  ASAS. 

Given  these  lessons  learned,  several  recommendations  for  the  ET 
design  guidelines  can  be  suggested.  First,  for  an  emerging  system  such 
as  ASAS/ENSCE,  a  design  concept  should  be  developed  that  is  as  detailed 
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as  possible  as  early  as  possible,  Couching  on  all  issues  that  pertain 
to  the  ET  component  of  a  training  system.  In  cases  in  which  infor¬ 
mation  is  lacking,  assumptions,  which  are  clearly  labeled  as  such,  may 
be  made  in  order  to  produce  a  complete  concept.  A  revised  package 
should  be  developed  as  soon  as  there  are  data  to  support  such  an 
effort.  The  design  concept  should  contain  a  discussion  of  the  training 
issues  which  will  be  addressed  and  augmented  in  future  revisions.  This 
initial  design  concept  would  give  guidance  for  decision  making  with 
regard  to  the  contents  of  training  when  ET  is  actually  produced.  The 
variables  and  issues  that  are  addressed  in  the  design  concept  should 
reflect  those  that  appear  in  the  existing  ET  design  documents,  but 
could  be  supplemented  by  others  that  are  selected  from  similar 
documentation  or  which  can  be  supported  in  some  way. 

A  second  recommendation  for  the  development  of  ET  design  guide¬ 
lines  concerns  the  state  of  explication  in  the  current  procedures.  The 
guidelines  require  expansion  in  the  guidance  they  supply  for  decision 
making  concerning  the  selection  of  values  or  contents  for  training 
variables. 


ASAS  ET  Integration 


The  task  of  developing  an  ET  integration  concept  for  ASAS  resulted 
in  two  lessons.  First,  it  was  found  that  due  to  the  lack  of  system 
stability,  it  was  impossible  to  develop  a  detailed  integration  concept 
for  ASAS  ET  without  making  assumptions  about  the  operational  system. 

The  major  difficulty  that  arose  was  that  the  system  interface  is  still 
being  modified.  However,  statements  concerning  integration  were  made 
based  on  available  information  and  assumptions. 

The  second  difficulty  that  was  experienced  during  the  development 
of  the  integration  concept  was  a  lack  of  detailed  knowledge  concerning 
the  operational  software  for  the  system.  Without  that  information,  it 
was  difficult  to  pinpoint  the  specific  "hooks"  between  the  training 
software  to  be  developed  and  the  operational  software.  By  "hooks"  it 
is  meant  both  specif’ c  interface  components,  at  the  level  of  individual 
screens  and  input  data  required  of  the  learner,  and  the  aspects  of  the 
computer  code  that  allows  for  the  training  software  to  stimulate  the 
operational  software.  Although  one  can  suggest  alternative  ways  for 
the  training  and  operational  software  to  interact,  these  "hooks"  can 
really  only  be  specified  if  the  form  of  all  potential  user-system 
interactions  is  known  and  if  the  operational  software  is  understood  at 
a  detailed  level.  For  example,  in  order  to  train  a  user  on  word 
processing  capabilities  of  a  system,  one  must  know  the  format  of  data 
presented  on  the  screen,  the  available  commands,  the  alternatives  for 
eitering  commands,  the  results  of  those  behavioral  alternatives  and  the 
iuput  commands,  typical  contexts  for  text  processing,  and  the  way  in 
which  one  can  integrate  the  text  processing  software  code  with  the 
training  software  code.  In  the  case  of  ASAS/ENSCE,  however,  very 
little  of  the  information  available  was  at  the  level  appropriate  for 
determining  specific  "hooks."  This  constraint  limited  the  type  of 
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integration  information  that  could  be  produced.  Therefore  only  the 
categories  of  operational  software  to  be  stimulated  and  their  contexts 
for  stimulation  were  discussed. 


Summary  for  ET  Design  and  Integration 

Thus  for  an  emerging  system,  ET  design  and  integration  concepts 
must  be  based  on  a  combination  of  fact  and  informed  fiction. 

Initially,  the  design  and  integration  concepts  should  be  very  detailed, 
but  primarily  based  on  assumptions.  In  fact,  it  might  be  useful  to 
generate  multiple  sets  of  concepts,  each  one  based  on  a  particular  set 
of  well-defined  assumptions.  As  details  of  the  operational  system 
become  available,  the  ET  component  concepts  can  be  revised  to  reflect 
these  changes,  moving  the  concepts  from  the  fiction  end  of  the 
continuum  to  the  "all  fact"  end. 


CONCLUSIONS 


Through  all  phases  of  Che  project,  two  problems  stand  out.  Each 
of  these  problems  affects  the  way  In  which  one  approaches  training 
design  for  emerging  systems. 

First  of  all,  for  an  emerging  system  such  as  ASAS/ENSCE,  there  are 
very  few,  if  any,  SMEs  available  to  offer  input  into  the  training 
development  process.  This  lack  of  SME  support  is  especially  true  in 
the  case  of  tasks  with  a  large  cognitive  component. 

A  second  problem  encountered  during  all  phases  of  this  project  was 
the  fluctuating  nature  of  the  system  during  the  development  phase. 

These  fluctuations  resulted  in  documentation  that  was  both  incomplete 
and  invalid  in  its  details. 

These  two  problems  may  be  overcome  by  the  use  of  two  procedures. 
The  utilization  of  these  procedures  will  not  necessarily  produce  a 
completely  valid  design  for  ET  at  an  early  stage  in  operational 
development,  but  will  lay  the  groundwork  for  the  generation  of  a  valid 
ET  component  design  at  some  later  time. 

During  early  stages  of  the  development  of  the  operational  system, 
an  ET  design  should  be  developed  that  is  based  on  a  combination  of 
available  data  and  assumptions  that  "flesh  out"  the  existing  infor¬ 
mation.  As  new  data  become  available,  the  design  should  be  modified  to 
incorporate  them.  Another  alternative  is  to  develop  a  set  of  ET  design 
concepts,  each  based  on  a  different  set  of  assumptions  in  combination 
with  verified  data.  As  the  operational  system  develops,  the  design 
concept  that  is  closest  to  the  factual  situation  would  be  the  one 
selected  for  modification  and  use  for  the  future  iterations. 

A  second  procedure  that  seems  to  offer  utility  in  the  development 
of  an  ET  design  is  the  flagging  of  unvalidated  information.  For  this 
project,  the  flagging  procedure  consisted  of  the  assignment  of 
certainty  ratings  to  values  for  complexity  and  criticality  for  each 
task.  Assigning  an  indicator  of  certainty  allows  one  to  differentiate 
between  uninformed  assumptions,  valid  data,  and  those  partially 
informed  "guesses"  or  assumptions  that  fall  somewhere  in  the  middle  of 
the  continuum  between  the  two  extremes. 

Finally,  the  thread  of  thought  that  connects  the  two  previous 
ideas,  the  design  of  the  ET  component  of  a  training  system  must  be 
viewed  as  an  iterative  process.  As  the  data  are  made  available,  task 
Lists,  ETRs,  and  the  ET  component  design  all  must  be  updated.  The  ET 
component  design  is  also  affected  by  operational  system  hardware  and 
software  constraints,  and  by  parameters  imposed  by  others  involved  in 
the  development  of  the  operational  system  and  the  training  for  that 


system.  The  best  way  to  ensure  that  the  designers  of  the  operational 
system  will  take  into  account  the  software  and  hardware  requirements  of 
the  ET  system  is  for  the  training  developers  to  produce  a  detailed 
training  design  that  includes  ET  estimates  at  the  earliest  point  as 
possible  in  operational  system  development.  Such  estimates  will  give 
the  developers  of  the  operational  system  some  basis  for  making  design 
judgments  that  will  affect  the  incorporation  of  ET  into  the  operational 
system. 
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